Why Programming-By-Demonstration Systems Fail: Lessons Learned for Usable AI

enable ordinary end users to create programs without needing to learn the arcane details of programming languages, but simply by demonstrating what their program should do. If PBD were successful, the vast population of nonprogrammer computer users would be able to take control of their computing experience and create programs to automate routine tasks, develop applications for their specific needs, and manipulate information in service of their goals.1 However, PBD has yet to achieve widespread adoption, partly because the problem is extremely difficult. How can any system successfully guess the user’s intended program out of an infinite space of possible programs? PBD is a natural match for artificial intelligence, particularly machine learning. By observing the actions taken by the user (training examples), the system can create a program (learned model) that is able to automate the same task in the future (predict future behavior). However, unlike most machine-learning systems that can rely on hundreds or thousands of training examples, users are rarely willing to provide more than a handful of examples from which the system can generalize. This constraint makes the design of machine-learning algorithms for PBD extremely challenging: they must learn accurately from an absurdly small number of user-provided training examples. However, when designing machine-learning algorithms for use in a user-facing system, accuracy is not the only important factor. Researchers’ experience designing and deploying machine-learning-based PBD systems reveals several factors that prevent users from wanting to use such systems. This paper presents some of the lessons researchers have learned about making AI systems usable.